{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Chapter_09",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jo-cho/advances_in_financial_machine_learning/blob/master/Chapter_09.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6F_F36iIJVj5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KWok5fVOJjYJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n",
        "from sklearn.ensemble import BaggingClassifier\n",
        "from sklearn.svm import SVC\n",
        "from sklearn import metrics"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RSNVf92qJtYT",
        "colab_type": "text"
      },
      "source": [
        "Exercises"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B65PWFhqJurf",
        "colab_type": "text"
      },
      "source": [
        "# 9.1. Using the function getTestData from Chapter 8, form a synthetic dataset of 10,000 observations with 10 features, where 5 are informative and 5 are noise."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aW_02quPKlDV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def getTestData(n_features=40,n_informative=10,n_redundant=10,n_samples=10000):\n",
        "  # generate a random dataset for a classification problem\n",
        "  from sklearn.datasets import make_classification\n",
        "  trnsX,cont = make_classification(n_samples=n_samples,n_features=n_features,n_informative=n_informative,n_redundant=n_redundant,random_state=0,shuffle=False)\n",
        "  df0 = pd.DatetimeIndex(periods=n_samples,freq=pd.tseries.offsets.BDay(), end=pd.datetime.today())\n",
        "  trnsX,cont = pd.DataFrame(trnsX,index=df0), pd.Series(cont,index=df0).to_frame('bin')\n",
        "  df0 = ['I_'+str(i) for i in range(n_informative)]+ ['R_'+str(i) for i in range(n_redundant)]\n",
        "  df0+=['N_'+str(i) for i in range(n_features-len(df0))]\n",
        "  trnsX.columns=df0\n",
        "  cont['w']=1./cont.shape[0]\n",
        "  cont['t1']=pd.Series(cont.index,index=cont.index)\n",
        "  return trnsX,cont"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kQAnLNfWKmCE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "debaa46f-a54a-4645-ea90-7c6b11a4e22c"
      },
      "source": [
        "trnsX, cont = getTestData(10,5,0,3000)"
      ],
      "execution_count": 135,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: FutureWarning: Creating a DatetimeIndex by passing range endpoints is deprecated.  Use `pandas.date_range` instead.\n",
            "  \"\"\"\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VFLzcTl5OsrV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "y= cont[['bin']]\n",
        "t1=cont['t1']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kAE3uX-KJ5K0",
        "colab_type": "text"
      },
      "source": [
        "## (a) Use GridSearchCV on 10-fold CV to find the C, gamma optimal hyperparameters on a SVC with RBF kernel, where param_grid={'C':[1E-2,1E-1,1,10,100],'gamma':[1E-2,1E-1,1,10,100]} and the scoring function is neg_log_loss."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ETx3TT7sTs5",
        "colab_type": "text"
      },
      "source": [
        "Grid search cross-validation conducts an exhaustive search for the combination of parameters that maximizes the CV performance, according to some user-defined\n",
        "score function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CeEu-W6FLCMU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "30f04bbe-7eab-414f-f228-506eb0112b92"
      },
      "source": [
        "#!pip install -q mlfinlab"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[?25l\r\u001b[K     |██▊                             | 10kB 22.9MB/s eta 0:00:01\r\u001b[K     |█████▌                          | 20kB 1.4MB/s eta 0:00:01\r\u001b[K     |████████▎                       | 30kB 1.3MB/s eta 0:00:01\r\u001b[K     |███████████                     | 40kB 1.6MB/s eta 0:00:01\r\u001b[K     |█████████████▉                  | 51kB 1.8MB/s eta 0:00:01\r\u001b[K     |████████████████▋               | 61kB 2.2MB/s eta 0:00:01\r\u001b[K     |███████████████████▍            | 71kB 1.9MB/s eta 0:00:01\r\u001b[K     |██████████████████████▏         | 81kB 2.1MB/s eta 0:00:01\r\u001b[K     |█████████████████████████       | 92kB 2.4MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▊    | 102kB 2.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▌ | 112kB 2.6MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 122kB 2.6MB/s \n",
            "\u001b[?25h"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZrhrBMjZLMR0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from mlfinlab import cross_validation\n",
        "from mlfinlab.cross_validation import PurgedKFold"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OVyXhbmSsK8L",
        "colab_type": "text"
      },
      "source": [
        "We need to pass our PurgedKFold class in order to prevent that GridSearchCV overfits the ML estimator to leaked information."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dS6t_ZlgLBaj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "n=5\n",
        "pkf = PurgedKFold(n, t1, pct_embargo=0.01)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w2MUJVZROxtL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "svc = SVC(kernel='rbf',probability=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UnUDVMrEuMm-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "param_grid = {'C':[1E-2,1E-1,1,10,100],'gamma':[1E-2,1E-1,1,10,100]}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y7tBYqitNEP8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gs0 = GridSearchCV(estimator = svc, param_grid = param_grid, cv = pkf, scoring='neg_log_loss',n_jobs=-1, iid=False, return_train_score=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zM51NQ1wN5lZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "30b8021c-3276-4bb6-9e60-9f47de998cc9"
      },
      "source": [
        "gsfit = gs0.fit(trnsX, y)"
      ],
      "execution_count": 180,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
            "  y = column_or_1d(y, warn=True)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3gccM7ldwM8G",
        "colab_type": "text"
      },
      "source": [
        "### About scoring:\n",
        "\n",
        "I advise you to use scoring='f1' in the context of meta-labeling applications,\n",
        "for the following reason. Suppose a sample with a very large number of negative (i.e.,label ‘0’) cases. A classifier that predicts all cases to be negative will achieve high 'accuracy' or 'neg_log_loss', even though it has not learned from the features how to discriminate between cases. In fact, such a model achieves zero recall and undefined precision . The 'f1' score corrects for that performance inflation by scoring the classifier in terms of precision and recall.\n",
        "\n",
        "For other (non-meta-labeling) applications, it is fine to use 'accuracy' or 'neg_log_loss', because we are equally interested in predicting all cases. Note that a relabeling of cases has no impact on 'accuracy' or 'neg_log_loss' however it will have an impact on 'f1'."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R8La9FI0b8Ec",
        "colab_type": "text"
      },
      "source": [
        "The key conceptual difference between accuracy and negative log-loss is that\n",
        "negative log-loss takes into account not only whether our predictions were correct\n",
        "or not, but the probability of those predictions as well."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FNyu_Exg0AuJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gsresult = pd.DataFrame(gsfit.cv_results_)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lY0_5zMz5vOr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "outputId": "7d7bf2ef-c728-49dc-f1b3-b208f963db9a"
      },
      "source": [
        "gsresult.columns"
      ],
      "execution_count": 182,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['mean_fit_time', 'std_fit_time', 'mean_score_time', 'std_score_time',\n",
              "       'param_C', 'param_gamma', 'params', 'split0_test_score',\n",
              "       'split1_test_score', 'split2_test_score', 'split3_test_score',\n",
              "       'split4_test_score', 'mean_test_score', 'std_test_score',\n",
              "       'rank_test_score', 'split0_train_score', 'split1_train_score',\n",
              "       'split2_train_score', 'split3_train_score', 'split4_train_score',\n",
              "       'mean_train_score', 'std_train_score'],\n",
              "      dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 182
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FO3LR5oH8U42",
        "colab_type": "text"
      },
      "source": [
        "## (c)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7cfoKv1FzSkA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "42a1351e-6dd2-430e-a976-3f48ac29a1fd"
      },
      "source": [
        "len(gsresult)\n",
        "# 5x5"
      ],
      "execution_count": 183,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "25"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 183
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TDKC0Aot8Yqx",
        "colab_type": "text"
      },
      "source": [
        "## (d)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BjNDjclF6PmZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "cc2f8b24-fc28-48c1-ffc8-c81028f36cbd"
      },
      "source": [
        "gsresult['mean_fit_time'].sum()"
      ],
      "execution_count": 184,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "45.315320062637326"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 184
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JxEW9tr46cYc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "96fd7256-188b-42dc-f54f-4ed6e0470f9f"
      },
      "source": [
        "gsresult['mean_score_time'].sum()"
      ],
      "execution_count": 185,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1.5529106616973876"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 185
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_CHNCmKY8bqB",
        "colab_type": "text"
      },
      "source": [
        "## (e)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "clwkvUOE6Gcb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "217df383-1ccd-4ba0-8522-8b4d01caab26"
      },
      "source": [
        "gsfit.best_params_"
      ],
      "execution_count": 186,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'C': 10, 'gamma': 0.1}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 186
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rW_LTRro9Cv_",
        "colab_type": "text"
      },
      "source": [
        "## (f)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C_klSqx78gOR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        },
        "outputId": "fdfd5cf6-ecd1-4319-b18b-e9e926495c9d"
      },
      "source": [
        "gsresult.sort_values('rank_test_score').head(1)[['rank_test_score','mean_test_score','mean_train_score']]"
      ],
      "execution_count": 187,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>rank_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>mean_train_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>1</td>\n",
              "      <td>-0.340678</td>\n",
              "      <td>-0.053253</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    rank_test_score  mean_test_score  mean_train_score\n",
              "16                1        -0.340678         -0.053253"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 187
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v_jWFeNV6oxe",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "59ffbfb4-11f9-43ff-aa60-67eaa3375acf"
      },
      "source": [
        "gsfit.best_score_"
      ],
      "execution_count": 188,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "-0.34067817387878235"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 188
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YqJF_jX08Kn9",
        "colab_type": "text"
      },
      "source": [
        "## (g)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lP1JubWO6s8c",
        "colab_type": "text"
      },
      "source": [
        "This example introduces nicely one limitation of sklearn’s Pipelines : Their\n",
        "fit method does not expect a sample_weight argument. Instead, it expects a\n",
        "fit_params keyworded argument. That is a bug that has been reported in GitHub;\n",
        "however, it may take some time to fix it, as it involves rewriting and testing much functionality. Until then, feel free to use the workaround in Snippet 9.2. It creates a newclass, called MyPipeline,which inherits allmethods from sklearn’s Pipeline. It overwrites the inherited fit method with a new one that handles the argument\n",
        "sample_weight, after which it redirects to the parent class.\n",
        "\n",
        "-------------\n",
        "```\n",
        "#Snippet 9.2\n",
        "\n",
        "class MyPipeline(Pipeline):\n",
        "  def fit(self,X,y,sample_weight=None,**fit_params):\n",
        "    if sample_weight is not None:\n",
        "      fit_params[self.steps[-1][0]+'__sample_weight']=sample_weight\n",
        "    return super(MyPipeline,self).fit(X,y,**fit_params)\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZErBZz7B9HAz",
        "colab_type": "text"
      },
      "source": [
        "#2."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YwNX7qFczJuf",
        "colab_type": "text"
      },
      "source": [
        "For ML algorithms with a large number of parameters, a grid search cross-validation\n",
        "(CV) becomes computationally intractable. In this case, an alternative with good statistical\n",
        "properties is to sample each parameter from a distribution"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H36lPYL9xfKh",
        "colab_type": "text"
      },
      "source": [
        "## (a) \n",
        "Use RandomizedSearchCV on 10-fold CV to find the C,\n",
        "gamma optimal hyper-parameters on an SVC with RBF kernel,\n",
        "where param_distributions={'C':logUniform(a=1E-2,b=\n",
        "1E2),'gamma':logUniform(a=1E-2,b=1E2)},n_iter=25 and\n",
        "neg_log_loss is the scoring function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Aq_fhxw0-Lbb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from scipy.stats import rv_continuous,kstest\n",
        "class logUniform_gen(rv_continuous):\n",
        "# random numbers log-uniformly distributed between 1 and e\n",
        "  def _cdf(self,x):\n",
        "    return np.log(x/self.a)/np.log(self.b/self.a)\n",
        "def logUniform(a=1,b=np.exp(1)):\n",
        "  return logUniform_gen(a=a,b=b,name='logUniform')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M-J_kIK8-GNj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "param_distributions={'C':logUniform(a=1E-2,b= 1E2),'gamma':logUniform(a=1E-2,b=1E2)}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tmLz1PW79i_l",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "rs0 = RandomizedSearchCV(svc,param_distributions,n_iter=25,n_jobs=-1,iid=False,return_train_score=True, scoring='neg_log_loss', cv=n)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7pq2b2rs--Dt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "3cbdd972-c206-4159-95c3-ba4b293bbab2"
      },
      "source": [
        "rsfit = rs0.fit(trnsX,y)"
      ],
      "execution_count": 223,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
            "  y = column_or_1d(y, warn=True)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HM7EbqM7DNRW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "rsresult = pd.DataFrame(rsfit.cv_results_)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4NO_WrmrHQyl",
        "colab_type": "text"
      },
      "source": [
        "##(b)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T3aVfFG9_cUk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "07df1b5e-d176-4f67-d163-2a1448fe482a"
      },
      "source": [
        "rsresult['mean_fit_time'].sum()"
      ],
      "execution_count": 225,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "37.067204332351686"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 225
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J8QnDf94G6JJ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "6ad67c84-b2fa-482b-d4e3-536e0151f3c7"
      },
      "source": [
        "rsresult['mean_score_time'].sum()"
      ],
      "execution_count": 226,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1.2519288539886475"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 226
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5wajBtAWG-4N",
        "colab_type": "text"
      },
      "source": [
        "더 적은 시간 걸림"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wxg5pidQHSZx",
        "colab_type": "text"
      },
      "source": [
        "##(c)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rHve2sK4G8qf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "bed02114-b8a1-4e16-fa79-93a6926ff5a6"
      },
      "source": [
        "rsfit.best_params_"
      ],
      "execution_count": 227,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'C': 3.6091960984355933, 'gamma': 0.10004345852454348}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 227
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y85TY_qVZa2e",
        "colab_type": "text"
      },
      "source": [
        "꽤다름"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H5dKk6eFHbbu",
        "colab_type": "text"
      },
      "source": [
        "##(d)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QpaJQV5qHL1O",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        },
        "outputId": "bfecdab8-a1ad-4b78-b52c-fb223459053f"
      },
      "source": [
        "rsresult.sort_values('rank_test_score').head(1)[['rank_test_score','mean_test_score','mean_train_score']] #this is neg-log-loss"
      ],
      "execution_count": 228,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>rank_test_score</th>\n",
              "      <th>mean_test_score</th>\n",
              "      <th>mean_train_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>1</td>\n",
              "      <td>-0.190361</td>\n",
              "      <td>-0.074954</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    rank_test_score  mean_test_score  mean_train_score\n",
              "10                1        -0.190361         -0.074954"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 228
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cfoeQhweZYdp",
        "colab_type": "text"
      },
      "source": [
        "더좋음"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aw5I2rXHPxcT",
        "colab_type": "text"
      },
      "source": [
        "#3. \n",
        "from 1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "knebQA8vP6b0",
        "colab_type": "text"
      },
      "source": [
        "## (a) Compute the Sharpe ratio of the resulting in-sample forecasts, from point 1.a (see Chapter 14 for a definition of Sharpe ratio)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "68jjzhw0P-Ji",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def sharpe(r):\n",
        "  s = r.mean()/r.std()\n",
        "  return s"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "El32vi2mQJx2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "cfe8c65a-7428-42f7-97f7-953f034bc2f9"
      },
      "source": [
        "#in-sample forecast\n",
        "\n",
        "pred3a = gsfit.best_estimator_.predict(trnsX)\n",
        "pred3a"
      ],
      "execution_count": 214,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0, 0, 0, ..., 1, 1, 1])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 214
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tv9sOm60Zfuc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "4b515951-5444-4168-fcc9-d9b4dc4a891c"
      },
      "source": [
        "#where is return in the data?\n",
        "#마음대로 가공해보자 bin=0이면 r=0, bin=1이면 실제에 따라 -1 or 1이라고 하자. 즉 실제로 bin=0인데 1로 했다면(틀리면) r=-1. (맞으면) r=1\n",
        "#실제는 cont['bin']\n",
        "\n",
        "r3a=pred3a*(2*cont['bin']-1)\n",
        "\n",
        "sharpe(r3a)"
      ],
      "execution_count": 215,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.948038314210756"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 215
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ohkof68OfDre",
        "colab_type": "text"
      },
      "source": [
        "##  (b)  Repeat point 1.a, this time with accuracy as the scoring function. Compute the in-sample forecasts derived from the hyper-tuned parameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RLYZBhUrcnrX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gs1 = GridSearchCV(estimator = svc, param_grid = param_grid, cv = pkf, scoring='accuracy',n_jobs=-1, iid=False, return_train_score=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "20u16wa-eFjT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "b522796e-15c1-46db-c7f6-410bd1992bfa"
      },
      "source": [
        "gsfit2 = gs1.fit(trnsX,y)"
      ],
      "execution_count": 206,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
            "  y = column_or_1d(y, warn=True)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UTteam57eJUQ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "10ffbe29-951f-408c-b1f5-dc5a3b2b0a8b"
      },
      "source": [
        "pred3b = gsfit2.best_estimator_.predict(trnsX)\n",
        "r3b = pred3b*(2*cont['bin']-1)\n",
        "sharpe(r3b)"
      ],
      "execution_count": 216,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.948038314210756"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 216
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uJ5YFKyGe-WS",
        "colab_type": "text"
      },
      "source": [
        "똑같음."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8tqsPMMRfKjQ",
        "colab_type": "text"
      },
      "source": [
        "# 4."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jjKW64MAfOZ8",
        "colab_type": "text"
      },
      "source": [
        "##(a) Compute the Sharpe ratio of the resulting in-sample forecasts, from point 2.a."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lOU0uOlUeemM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8b3805e2-90b9-4653-e84a-76c9f5d2d76e"
      },
      "source": [
        "pred4a = rsfit.best_estimator_.predict(trnsX)\n",
        "r4a = pred4a*(2*cont['bin']-1)\n",
        "sharpe(r4a)"
      ],
      "execution_count": 229,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.906000957368751"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 229
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PpsI7vZif8B0",
        "colab_type": "text"
      },
      "source": [
        "##(b)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vYZmK0hEfR5m",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "rs1 = RandomizedSearchCV(svc,param_distributions,n_iter=25,n_jobs=-1,iid=False,return_train_score=True, scoring='accuracy', cv=n)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Eooq5lZegQL6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "2072b2f8-2cc4-48b0-8f08-c8986dbb9eb6"
      },
      "source": [
        "rsfit2 = rs1.fit(trnsX,y)"
      ],
      "execution_count": 231,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
            "  y = column_or_1d(y, warn=True)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fCTPEKHmgSH4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "ba2a34ab-7662-47e4-9599-5f65dffdbbf1"
      },
      "source": [
        "pred4b = rsfit2.best_estimator_.predict(trnsX)\n",
        "r4b = pred4b*(2*cont['bin']-1)\n",
        "sharpe(r4b)"
      ],
      "execution_count": 232,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9383503729346667"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 232
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5GQ1LnePhkOa",
        "colab_type": "text"
      },
      "source": [
        "이건 또 다름"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TdMrpMazgb9J",
        "colab_type": "text"
      },
      "source": [
        "#5. Read the definition of log loss, L [Y, P]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t5j2qzf_kHtF",
        "colab_type": "text"
      },
      "source": [
        "https://towardsdatascience.com/understanding-negative-log-loss-8c3e77fafb79"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EbnIxk7JgciR",
        "colab_type": "text"
      },
      "source": [
        "Logarithmic loss (related to cross-entropy) measures the performance of a classification model where the prediction input is a probability value between 0 and 1. The goal of our machine learning models is to minimize this value. A perfect model would have a log loss of 0. Log loss increases as the predicted probability diverges from the actual label. So predicting a probability of .012 when the actual observation label is 1 would be bad and result in a high log loss."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j1on780Lkc3l",
        "colab_type": "text"
      },
      "source": [
        "http://wiki.fast.ai/index.php/Log_Loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xDZu-jOwkxu5",
        "colab_type": "text"
      },
      "source": [
        "Why negative?\n",
        "\n",
        "Log Loss uses negative log to provide an easy metric for comparison. It takes this approach because the positive log of numbers < 1 returns negative values, which is confusing to work with when comparing the performance of two models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WpwEWFIBlKxE",
        "colab_type": "text"
      },
      "source": [
        "# 6.Consider an investment strategy that sizes its bets equally, regardless of the forecast’s confidence. In this case, what is a more appropriate scoring function for hyper-parameter tuning, accuracy or cross-entropy loss?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W6g8oytYlfFn",
        "colab_type": "text"
      },
      "source": [
        "The key conceptual difference between accuracy and negative log-loss is that negative log-loss takes into account not only whether our predictions were correct or not, but the probability of those predictions as well"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lXw2oXUbmfKC",
        "colab_type": "text"
      },
      "source": [
        "위에 4번예제가 그런 경우라고 볼 수 있다. Accuracy로 점수를 매겨 파라미터 튜닝을 했을 때 더 높은 샤프ratio가 나왔다. 즉, 베팅사이즈를 1과 0으로만 생각한다면 accuracy가 더 괜찮을 수 있다. 예측이 맞냐틀리냐만 따지면 되기 때문이다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fdvVxQEmkhtq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}